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Preface 


The  Alberta  Heritage  Foundation  for  Medical  Research  (AHFMR)  health  technology 
assessment  initiative  series,  commenced  in  March  2000  with  “A  Framework  for 
regional  health  authorities  to  make  optimal  use  of  health  technology  assessment.” 
The  purpose  of  this  series  is  to  provide  policy  and  decision-makers  with  the  best 
informaton  available  on  howto  redesign  their  healthcare  structures  and  processes 
to  effectively  improve  the  outcomes  of  their  healthcare  delivery  systems.  This  paper 
undertakes  to  introduce  and  describe  an  innovative  approach  to  facilitating  patient 
safety  - risk  assessment  and  risk  management  analysis.  Although  their  subject  of 
study  is  radiotherapy  the  authors  encourage  the  application  of  the  appaoch  to  other 
healthcare  interventions. 

Other  papers  in  this  series  are  listed  on  the  inside  cover.  Copies  of  these  and 
other  reports  can  be  found  at:  http://www.ahfmr.ab.ca/frames3.html 

If  you  have  any  comments  or  suggestions  to  make  on  this  paper, 

II  would  be  delighted  to  receive  your  feedback. 

Don  Juzwishin 

Director 

Health  Technology  Assessment 
Alberta  Heritage  Foundation 
for  Medical  Research 
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Introduction 


“First,  do  no  harm.”  Yet,  patient  harm  does  occur  in  healthcare  , despite 
the  best  intentions  of  providers  and  healthcare  systems.  But,  how  do  we 
determine  whether  or  to  what  degree  a healthcare  provider  or  system 
is  harming  patients?  The  US  Institute  of  Medicine’s  report  of  44,000  to 
98,000  deaths  per  year  in  the  US  attributable  to  medical  errors  (1),  and 
recent  estimates  for  Canada  of  approximately  185,000  adverse  events  per 
year  (2)  are  staggering.  How  does  a healthcare  organization  effectively 
assess  and  manage  medical  incidents  in  which  patients  are  subjected 
to  deviations  from  beneficial  pathways  of  care?  Further,  as  innovation, 
dissemination,  and  implementation  of  medical  technologies  increase  at 
an  exponential  rate,  the  potential  exists  for  new  and  highly  uncertain  risks 
to  also  be  introduced  into  healthcare  at  an  exponential  rate.  An  implicit 
assumption  is  often  that  new  technologies  somehow  confer  a net  increase 
in  benefit  to  patients  and  organizations  simply  because  they  are  “new  and 
improved.”  This  is  not  necessarily  the  case. 

We  use  the  term  incident  because  some  of  these  deviations  may  result  in 
harm  to  patients,  and  some  may  not;  yet  analysis  of  all  can  be  informative. 
We  are  concerned  with  pathways  of  care,  because  patients  are  often 
subjected  to  multiple  treatments  from  multiple  providers,  sometimes  under 
multiple  organizations.  Further,  many  medical  treatments  result  in  harm, 
but  the  benefits  outweigh  the  harm.  For  example,  taking  aspirin  relieves 
a headache,  yet  may  result  in  stomach  bleeding.  Determination  of  the  net 
benefit  of  a pathway  of  care  or  technology,  whether  or  not  incidents  occur, 
is  clearly  important.  The  appropriate  balance  of  benefit  and  harm  becomes 
more  critical  and  difficult  to  identify  with  complex  treatments  such  as 
cancer  care.  Choices  are  further  complicated  because  resources  are  limited 
in  healthcare  , so  the  determination  of  the  net  benefit  of  a pathway  of  care 
or  technology  under  resource  constraints  is  particularly  difficult. 

There  is  no  doubt  that  progress  has  been  made  in  illuminating  some 
patient  safety  issues  and  possible  interventions;  this  qualitative  work 
is  the  subject  of  recent  reviews  (3-5).  There  is  general  agreement  that 
proactive,  transparent  approaches  to  patient  safety  (as  opposed  to  reactive, 
“sweep-under-the-rug”  historical  approaches)  are  not  only  desirable,  but 
essential.  In  this  paper  we  argue  that  rigorous,  quantitative  approaches 
used  in  areas  outside  medicine  involving  human  health  risks  such  as  nuclear 
safety,  environmental  regulation  and  food  safety  are  not  only  applicable 
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and  informative  in  healthcare,  but  will  likely  meet  the  requirements  of  a 
comprehensive  and  appropriate  safety  system  for  use  in  healthcare. 

We  argue  that  such  approaches  are  desirable  from  several  aspects,  as 
discussed  in  detail  later  in  this  paper,  but  in  general  the  advantages  include: 


The  intent  of  our  project  is 
to  “raise  the  bar”  in  terms 
of  how  decisions  ore  made 
regarding  resource  allocation 
for  clinical  risk  management. 


© They  are  concordant  with  evidence  based  decision-making  (EBDM),  as 
practiced  in  clinical  care; 

® They  address  multiple  objectives  of  multiple  stakeholders; 

« They  address  tradeoffs  across  the  benefits,  risks,  and  costs  associated 
with  different  medical  care  alternatives; 

• They  address  changes  in  technology  that  may  be  associated  with 
increased  or  decreased  potential  for  risk,  and  associated  costs. 

This  paper  summarizes  the  first  steps  in  a research  program  designed 
to  inform  technology  assessment  and  decision-making  associated  with 
radiotherapy  (treatment  of  cancer  with  ionizing  radiation)  at  the  Tom  Baker 
Cancer  Centre  (TBCC)  in  Calgary,  Alberta.  The  TBCC  is  a comprehensive 
cancer  care,  education  and  research  facility  that  is  part  of  the  Alberta 
Cancer  Board  (ACB)  and  closely  associated  with  the  University  of  Calgary. 
The  intent  of  our  project  is  to  “raise  the  bar”  in  terms  of  how  decisions 
are  made  regarding  resource  allocation  for  clinical  risk  management.  The 
TBCC  serves  as  a field  test  for  an  ongoing  research  program  approach  that 
may  have  general  applicability  in  many  areas  of  medicine. 
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Research  Team 


The  project  started  with  a chance  encounter  in  2002  between  a risk  analyst 
(Lee)  and  a medical  physicist  (Dunscombe);  both  new  to  the  Faculty  of 
Medicine  at  the  University  of  Calgary.  Lee  had  previously  spent  many  years 
consulting  in  risk  analysis  and  risk  management  in  a variety  of  settings. 
Dunscombe,  although  primarily  a radiation  oncology  physicist,  had  also 
been  active  in  activity-based  costing  and  health  economics  in  clinical 
settings  for  several  years.  Soon  after  this  informal  collaboration  began,  two 
radiation  oncologists  (Kelly  and  Craighead)  joined  the  group.  Craighead, 
in  addition  to  his  clinically  oriented  research,  has  worked  in  the  area  of 
health  services  delivery.  More  recently,  an  additional  medical  physicist  and 
former  international  management  consultant  (Newcomb)  joined  the  team. 
Additional  content  experts  have  been  consulted  on  specific  aspects  of  the 
project. 

An  initial  major  challenge  was  securing  research  funding  for  the  project. 
Capacity  was  needed  to  help  the  team  collect  information  and  to  model  the 
system.  Because  many  of  the  approaches  that  we  desire  to  implement  are 
novel  to  healthcare,  we  did  not  expect  to  find  anyone  with  direct  experience 
in  such  approaches,  so  we  sought  an  individual  with  general  experience 
in  mathematical  modeling  and  decision  science  methods.  Funding  was 
secured  from  the  Canadian  Institutes  of  Health  Research,  and  an  engineer 
with  expertise  in  uncertainty  management  in  intelligent  systems  (Ekaette) 
was  hired.  Additional  research  funding  is  being  applied  for  on  a continuous 
basis  to  recruit  additional  staff  for  various  aspects  of  the  project.  For 
example,  a recent  Ph.D.  graduate  in  operations  management  with  many 
years  of  safety  management  experience  in  industry  (Cooke)  has  been 
recruited  for  a postdoctoral  fellowship  in  patient  safety.  This  project,  as 
well  as  other  similar  projects,  are  being  conducted  under  a larger  research 
program  called  the  Health  Technology  Research  Group  (HTRG);  part  of 
the  Department  of  Community  Health  Sciences  (Faculty  of  Medicine),  the 
Centre  for  Health  and  Policy  Studies,  and  the  Southern  Alberta  Cancer 
Research  Institute  at  the  University  of  Calgary. 
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Problem  Formulation 

3.1  Conceptual  process 


In  order  for  risks  to  be 
effectively  managed,  they 
must  be  quantified. 


understand  decision  situation  and  context 


define  the  question 


identify  objectives 


identify  alternatives 


decompose  and  model  the  problem 
(structure,  uncertainty,  preferences) 


rank  alternatives 


choose  the  best  alternative 


perform  sensitivity  analysis 


is  further  analysis  needed? 


implementation 


The  conceptual  process  that  we  are  following  is  illustrated  in  Figure  1.  This 
is  a generalized  framework  based  on  common  sense,  as  well  as  methods 
employed  in  statistical  decision  theory  and  strategic  planning  that  have 
been  well  established  since  the  1960s  (6).  The  key  to  this  process  is  that  it  is 
not  only  qualitative,  but  that  it  can  be  represented  by  a quantitative  model, 
in  the  spirit  of  the  following  maxim:  In  order  for  risks  to  be  effectively 
managed,  they  must  be  quantified.  This  is  a fact  that  is  recognized  in 
nearly  every  other  industry  (insurance,  financial  management,  industrial 

safety,  nuclear  safety,  aerospace, 
environmental  management;  to  name  a 
few),  but  remains  largely  unrealized  in 
healthcare. 


Risk  is  not  an  abstract  concept; 
rather,  it  is  defined  as  a function  of 
the  probability  and  severity  of  adverse 
events  (7-9).  If  risk  is  quantified,  then 
reduction  in  risk  by  means  of  risk 
management  can  be  measured  and 
tracked.  However,  risks  are  perceived 
differently  by  scientists  and  engineers 
as  compared  to  individuals  without 
quantitative  training;  thus,  proper 
communication  of  risk  is  critical  (10). 


YES 


Figure  i:  General  risk  analysis  and 
management  framework 


3.2  Patient  safety 
in  radiotherapy 

This  team  recognized  early  on  that 
quantification  of  risk  was  desirable, 
and  the  first  step  was  to  define  the 
decision  context  and  the  question. 
Radiotherapy  for  cancer  is  a critical 
medical  procedure  that  occurs  in 
a complex  environment  involving 
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numerous  health  professions,  complex  machines  and  hardware,  as 
well  as  software.  The  potential  exists  for  rare  incidents  that  can  lead  to 
inappropriate  administration  of  radiation  to  patients,  with  potentially 
catastrophic  consequences  such  as  premature  death  or  appreciably 
impaired  quality-of-life. 

Catastrophic  incidents  have  occurred  in  radiotherapy  in  some  jurisdictions 
(Table  i).  We  define  catastrophic  incidents  as  incidents  that  result  in 
patient  mortality  or  severe  loss  in  quality-of-life.  As  one  example,  in  Canada 
in  the  1980s,  Atomic  Energy  of  Canada  Limited  introduced  a radically 
new,  progressive  design  of  linear  accelerator  featuring  a large  degree  of 
computer  control.  A design  fault  resulted  in  lethal  burns  for  three  patients 
and  several  others  received  serious  radiation  misadministrations  (11-15). 
Incidents  involving  misadministration  can  also  occur  in  the  opposite 
direction;  i.e.  if  a patient  does  not  receive  adequate  therapy,  their  cancer 
can  progress  unnecessarily.  Based  on  media  observations  (11515),  multiple 
Table  1 : Examples  of  incidents  resulting  fatalities  or  injuries  are  of  greater  public  interest  than  individual  incidents, 

in  serious  injury  or  death  to  patients  and  thus  reported  incidents  can  be  expected  to  comprise  only  a fraction  of 
(references  11-15)  those  incidents  actually  occurring. 


Year 

Country 

Number  of 
Patients  Affected 

Cause 

Consequences 

1985-87 

Canada 

6 

Faulty  design 

Death,  morbidity 

00 

00 

England 

207 

Miscalibration 

Death,  morbidity 

1999 

Panama 

9 

Software  bug 

Death,  morbidity 

1988-99 

Japan 

254 

Technician  error 

Morbidity 

Although  quality  assurance/quality  control  (QA/QC)  mechanisms  are  in 
place  in  all  modern  cancer  treatment  facilities,  we  feel  there  is  still  room 
for  improvement.  In  particular,  we  see  an  increasing  need  to: 

• Predict  the  probability  of  such  incidents; 

• Determine  whether  current  QA/QC  mechanisms  are  truly  adequate  for 
risk  management  (i.e.  in  terms  of  reduction  of  probability/consequences 
of  catastrophic  events),  and  to  assess  alternative  mechanisms; 

• Determine  whether  resources  are  being  expended  in  the  most  efficient 
way  to  prevent  such  incidents,  while  providing  high  quality  patient 
therapy. 
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Thus,  the  fundamental  decision  objectives  or  goals  of  our  project  can  be 
represented  as: 

• Maximizing  patient  quality-of-care; 

• Minimizing  patient  harm; 

• Staying  within  resource  constraints. 

There  are  other  objectives  that  may  be  important,  such  as  maximizing 
provider  satisfaction,  minimizing  the  potential  for  legal  action,  and  the 
like;  but  at  the  beginning  of  the  risk  management  process  our  desire  has 
been  to  concentrate  on  the  above  risk  dimensions  of  the  problem. 
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Challenges  Associated  with  Patient 
Safety  Risk  Management  in  Healthcare 


Once  it  is  recognized  that  quantification  of  risk  is  desirable  and  a problem 
is  roughly  defined,  then  a challenge  arises  with  regard  to  information. 
Reliable  quantitative  risk  information  in  healthcare  is  often  lacking 
compared  with  other  fields  and  is  subject  to  large  biases(i;i6;i7).  So  how 
can  reduction  in  risk  be  determined?  One  way  forward  is  to  collect  data. 
This  is  a major  reason  for  current  efforts  to  implement  incident  reporting 
systems  and  electronic  health  records.  This  is  likely  a reasonable  and 
efficient  way  to  address  frequent  incidents  in  a retrospective  fashion, 
but  what  about  proactive  management  of  rare  incidents  that  result  in 
catastrophic  consequences  (e.g.  Table  i)?  Such  events  are  rare,  but 
particularly  as  they  may  involve  multiple  patients  and/or  catastrophic 
consequences  not  only  associated  with  the  patients  themselves,  but  with 
diversion  of  resources,  potential  litigation,  etc.,  the  need  for  management 
of  both  systematic  and  random  incidents  becomes  apparent.  A challenge 
associated  with  risk  management  of  rare  events  lies  in  prediction  of  these 
events,  even  if  they  have  not  occurred.  Standard  statistical  analyses  are 
obviously  of  little  utility  in  such  situations. 

An  additional  challenge  is  to  explicitly  evaluate  the  tradeoffs  associated 
with  risk  management  strategies.  All  risk  management  costs  something, 
be  it  time,  money,  human,  or  other  resources.  Resources  spent  on  risk 
management  may  be  better  spent  on  other  aspects  of  healthcare.  This 
becomes  particularly  important  in  a publicly  funded  healthcare  system, 
where  hospitals  or  regions  operate  under  a limited  budget  controlled  by 
government  agencies.  Indeed,  in  an  integrated  system  such  as  the  ACB 
or  the  Calgary  Health  Region  (CHR)  in  Alberta,  if  increased  resources  are 
spent  on  risk  management  in  one  healthcare  department  or  hospital,  then 
the  care  of  patients  in  another  hospital  or  component  of  the  healthcare 
system  may  be  compromised.  In  the  quality  management  field,  quality  is 
often  viewed  as  being  free,  as  expenditures  on  quality  improvement  are 
often  paid  back  many  times  by  the  value  realized.  It  is  unclear  whether  this 
is  the  case  in  healthcare  settings,  particularly  in  publicly  funded  systems. 
Research  of  the  type  described  here  is  likely  be  informative  in  this  regard. 
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From  another  perspective,  if  a healthcare  system  mandates  a risk 
management  strategy,  but  has  not  accounted  for  the  preferences  of 
providers,  legal  constraints,  union  issues,  and  the  like,  then  an  obviously 
suboptimal  situation  will  result.  Further,  many  risk  management  strategies 
impose  a new  set  of  risks.  For  example,  say  a healthcare  system  implements 
a new  technology,  such  as  an  electronic  health  record  system  with  a major 
goal  of  reducing  prescribing  incidents.  What  are  the  risks  associated  with 
the  electronic  system  itself  (e.g.  what  happens  when,  not  if,  it  crashes?). 
How  do  these  risks  compare  to  the  benefits  conferred  by  the  new 
technology,  and  how  can  the  risks  be  effectively  and  efficiently  managed? 
Figure  2:  Different  types  of  health  Standard  financial  and  economic  analyses  do  not  address  these  tradeoffs, 

technology  research  activities  and  are  thus  of  limited  utility  to  decision-makers. 
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The  overall  goal  of  our 
research  program  is  to 
provide  a quantitative 
methodology  that 
will  inform  QA/QC,  risk 
management,  and  health 
technology  assessment 
practices  in  healthcare. 


Risk  management  is  intimately  tied  with  technology  assessment.  For 
example,  Canadian  cancer  treatment  programs  are  considering  various 
forms  of  stereotactic  radiosurgery  facilities.  Although  a literature  review 
and  a costing  study  have  been  conducted  and  published  by  the  Alberta 
Heritage  Foundation  for  Medical  Research  (18-19),  such  new  technologies 
potentially  introduce  a new  set  of  benefits,  risks,  and  costs  into  a particular 
setting.  An  expanded  definition  of  and  role  for  technology  assessment  in 
healthcare  settings  may  include  the  types  of  activities  that  we  are  currently 
pursuing  through  the  University  of  Calgary  HTRG  (Figure  2),  which  cover 
the  continuum  illustrated. 

The  assessments  under  health  technology  assessment  typically  summarize  and 
synthesize  existing  technical  information  regarding  clinical  effectiveness 
and  financial  costs.  The  analyses  under  health  technology  appraisal 
incorporate  explicit  consideration  of  risk,  as  well  as  socioeconomic 
considerations  and  tradeoffs  across  different  objectives  or  goals  in 
decision-making.  The  activities  under  health  technology  implementation  are 
oriented  toward  evaluation  of  operational  considerations  and  portfolio 
management.  The  studies  under  health  technology  evaluation  are  outcomes 
oriented  and  relate  to  program  evaluation. 

Appreciable  challenges  exist  with  regard  to  EBDM  and  systems  thinking 
in  patient  safety,  but  these  challenges  are  not  insurmountable.  Innovative 
approaches  are  currently  being  applied  by  the  authors  to  address 
quantitative  risk  analysis  and  management  of  rare  events  in  radiotherapy; 
these  approaches  are  described  in  detail  below.  The  overall  goal  of  our 
research  program  is  to  provide  a quantitative  methodology  that  will  inform 
QA/QC,  risk  management,  and  health  technology  assessment  practices  in 
healthcare.  As  the  group’s  expertise  is  primarily  in  radiotherapy  we  have 
chosen  this  area  for  our  first  case  study. 

We  start  by  describing  some  other  approaches  that  have  been  applied  to 
patient  safety  in  medicine,  and  then  discuss  and  compare  our  approach  to 
these  other  approaches. 
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Current  Approaches  to  Patient  Safety 
Risk  Analysis  and  Management 


We  performed  a literature  review  to  determine  the  state-of-the-art  in  patient 
safety  risk  analysis  and  management.  The  primary  author  and  others 
previously  conducted  a review  of  risk  and  decision  analysis  methods  in 
healthcare  in  general;  the  interested  reader  should  consult  this  article  and 
the  references  therein  for  additional  information  on  the  state-of-the-art  in 
healthcare  evaluation  (20). 

PubMed  and  Internet  searches  using  the  terms  risk  assessment;  risk  analysis; 
decision  analysis;  patient  safety  and  medical  error  (no  date  limits)  identified 
over  200  documents,  for  which  abstracts  or  Web  summaries  were  reviewed. 
Based  on  our  review,  the  state-of-the-art  in  healthcare  risk  assessment  and 
risk  management  (aside  from  ad  hoc  approaches)  appears  to  be  represented 
by  approaches  such  as  failure  mode  and  effects  analysis  (FMEA)  (21), 
which  is  a semi-quantitative,  proactive  approach  with  engineering 
origins  currently  employed  by  providers  such  as  the  US  Veterans 
Administration  (22).  FMEA  qualitatively  characterizes  a system,  identifies 
potential  failure  points,  assigns  probability  and  severity  categories,  and 
incorporates  this  information  into  a decision  matrix/flowchart  (we  use 
the  term  semi-quantitative  because  FMEA  does  not  model  the  relationships 
between  system  variables,  and  does  not  explicitly  address  uncertainty). 

Other  qualitative/semi-quantitative  approaches  used  to  characterize 
potential  system  incidents  include  root  cause  analysis  (23)  and 
microsystems  analysis  (24),  but  neither  technique  quantitatively  estimates 
probability  and  severity,  and  both  are  typically  employed  reactively,  that 
is  after  an  event  has  occurred.  However,  these  approaches  and  FMEA  are 
reasonable,  important  steps  in  identifying  sources  of  potential  failures 
leading  to  adverse  patient  outcomes.  It  is  important  to  characterize 
human,  organizational,  technological,  and  information  management 
sources  of  failure,  because  they  are  the  key  to  identifying  risk  management 
alternatives  (9). 

Despite  their  utility,  the  methods  outlined  above  do  not  completely 
inform  a holistic  risk  assessment  and  management  strategy  in  complex, 
technologically  intensive  healthcare  systems  because  they  do  not: 

1.  Quantitatively  model  the  relationships  between  events  in  the  system; 
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2.  Identify  and  track  uncertainties  associated  with  the  system  itself  and 
system  variables; 

3.  Address  variability  across  patients,  provider  practice,  facilities,  etc.; 

4.  Address  rare  incidents; 

5.  Address  multiple  objectives  important  to  a healthcare  program,  and 
tradeoffs  across  objectives; 

6.  Address  resource  constraints; 

7.  Address  system  changes  such  as  new  equipment,  new  personnel,  etc.; 

8.  Address  operational  considerations. 

All  of  these  considerations  are  important  in  our  present  work.  However, 
fully  quantitative  risk  assessment  approaches  to  patient  safety  are  limited 
in  the  literature.  An  outstanding  exception  is  the  work  by  Pate-Cornell 
et  al.  (25526),  who  examined  anesthesia  related  risks  with  probabilistic 
risk  analysis  (PRA).  PRA  was  developed  in  engineering,  and  has  been 
applied  in  many  scenarios  ranging  from  nuclear  safety  to  environmental 
assessment  (758527).  Typically  using  simulation  techniques,  PRA  estimates 
the  probability  and  severity  of  adverse  events  by  identifying  potential 
incidents  and  propagating  the  uncertain  probability  of  failure  according  to 
system  characteristics. 

Uncertainty  analysis,  an  integral  component  of  PRA,  quantifies  what  is 
known  and  what  is  not  known  or  incompletely  known.  Uncertainty,  as 
used  here,  may  be  subjective  such  as  the  incomplete  knowledge  on  the 
part  of  an  expert  or  group  of  experts,  but  is  always  quantifiable  in  some 
sense.  Uncertainty  analysis  is  important  where  risk  attitude  is  relevant: 
for  instance  where  rapidly  increasing  aversion  to  increasing  levels  of 
risk  exists,  where  uncertain  information  from  different  sources  must 
be  combined,  and  where  decisions  must  be  made  regarding  additional 
resource  allocation  to  reduce  uncertainty  (28).  All  these  factor  into  risk 
management  decisions  involving  patient  safety.  Estimates  of  medical 
incident  rates  are  likely  to  be  undermined  by  large  uncertainties  and 
potential  biases  (16517),  but  this  has  not  been  fully  explored  in  the 
literature.  For  these  reasons  application  of  PRA  is  a useful  direction  in 
patient  safety  studies. 

While  PRA  is  useful  in  identifying  risks  and  associated  uncertainties,  it  does 
not  directly  guide  risk  management  decisions.  Decision  analysis  models  (6) 
can  be  helpful  in  this  respect,  particularly  because  healthcare  decisions 
involve  important  tradeoffs  (29).  Quantitative  decision  analyses  examine 
comparative  utility  across  different  strategies.  The  utility  of  a strategy  is  an 
aggregated  measure  of  how  well  that  strategy  attains  decision  objectives, 
which  are  quantitative  representations  of  individual  or  organizational 
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values  (30).  Examples  of  objectives  in  healthcare  are  maximizing  patient 
survival  and  quality-of-life,  minimizing  patient  harm,  maximizing 
compliance,  and  staying  within  resource  constraints.  Tradeoffs  across 
multiple  objectives  are  almost  inevitable  in  patient  safety  risk  management. 

Based  on  our  literature  search,  decision  analyses  addressing  medical 
incident  risk  management  in  healthcare  are  limited.  Many  decision 
analyses  of  pharmaceuticals  consider  adverse  effects  without  specifically 
addressing  patient  safety,  typically  using  cost-effectiveness  or  cost-utility 
ratios,  or  more  rarely,  net  benefit,  as  the  outcomes  (29).  Provider  safety  has 
been  addressed  by  Laufer  and  Chiarello  (31),  who  examined  interventions 
to  prevent  needlestick  injuries;  and  AuBuchon  and  Littenberg  (32),  who 
examined  cost-effectiveness  of  a mechanical  barrier  system  to  reduce 
mistransfusion  risk.  No  examples  of  true  multi-objective  decision  analyses 
addressing  medical  failure  scenarios  were  identified. 

Table  2:  Comparison  of  different  Table  2 compares  the  different  methods  discussed  here  as  well  as  others 

types  of  risk  assessment  and  risk  commonly  in  use,  incorporating  the  desirable  criteria  for  a risk  assessment 

management  analyses  and  management  framework  listed  above. 


METHOD 

CRITERION 

Ad  hoc 

Descriptive 

statistics 

RCA 

FMEA 

Microsystems 

modeling 

PRA 

Decision 

analysis 

Systematic 

- 

+ 

+ 

+ 

+ 

+ 

+ 

Proactive/predictive 

- 

- 

- 

+ 

- 

+ 

+ 

System-oriented 

- 

- 

+ 

+ 

+ 

+ 

+ 

Quantitative 

- 

+ 

- 

Semi  + 

Semi  + 

+ 

+ 

Addresses  parameter 
uncertainty 

- 

Semi  + 

- 

- 

- 

+ 

+ 

Addresses  variability 

- 

+ 

- 

- 

- 

+ 

+ 

Can  model  rare 
incidents 

- 

- 

- 

- 

- 

+ 

+ 

Addresses  multiple 
objectives 

- 

- 

- 

- 

- 

- 

+ 

Addresses  resource 
constraints 

- 

- 

- 

- 

- 

- 

+ 

Addresses  changes  to 
the  system 

- 

- 

- 

- 

- 

- 

+ 

+ = method  addresses  this  criterion  Notes  on  methods: 

- = method  does  not  address  this  criterion.  Ad  hoc:  although  different  criteria  may  be  met  by  an  ad  hoc  process, 

there  is  no  mechanism  for  ensuring  that  the  criteria  are  met. 

RCA:  root-cause  analysis. 

FMEA:  failure  mode  and  effects  analysis. 

PRA:  probabilistic  risk  analysis. 
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Figure  3:  Proposed  probabilistic  risk 
and  decision  analysis  (PRADA)  model 


RISK 

MANAGEMENT 

DECISION 


Probabilistic  Risk  and  Decision  Analysis 
(PRADA) 


Healthcare  providers  and  organizations  need  proactive,  systematic 
risk  management  strategies  that  incorporate  quantitative  risk  analysis, 
especially  for  complex  scenarios.  However,  few  examples  of  fully 
quantitative  approaches  to  patient  safety  risk  management  exist  in  the 
healthcare  literature,  and  none  were  located  that  combined  the  strengths 
of  the  powerful  methods  of  PRA  and  multi-attribute  decision  analysis. 
This  contrasts  with  work  in  industries  with  high  potential  public  health 
or  safety  impacts  (e.g.  the  nuclear  power  industry),  and  is  unfortunate 
because  quantitative  methods  potentially  add  so  much  value.  The  reasons 
for  the  limited  use  of  quantitative  approaches  are  unclear,  but  may 
relate  to  healthcare  being  typically  expert  or  practice  driven,  as  opposed 
to  systems-analysis  driven  (1).  Lack  of  training  in  quantitative  analysis 
beyond  descriptive  statistics  among  those  in  the  medical  field  may  also 
be  a contributing  factor.  Texts  such  as  Clemen  (6),  Cox  (7),  Haimes  (9), 
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and  others  referenced  here  provide  useful  methodologies  and  further 
references. 

PRA  can  potentially  characterize  highly  uncertain  probabilities  and 
consequences  that  influence  decision-making.  Based  on  the  experience 
of  the  authors  in  quantitative  methodologies,  we  believe  that  a holistic, 
multi-attribute  probabilistic  risk  and  decision  analysis  (PRADA)  model 
(Figure  3 on  the  previous  page)  could  potentially  inform  patient  safety 
risk  management  and  uncertainty  reduction  activities  in  technologically 
intensive  healthcare  institutions.  FMEA  and  similar  methods  may  serve 
as  useful  “front-ends”  for  such  quantitative  approaches  in  an  integrated 
strategy,  but  we  believe  that  adopting  rigorous,  quantitative  techniques 
could  advance  the  field  of  patient  safety  and  address  many  of  the 
shortcomings  of  existing  qualitative  assessment  frameworks.  To  our 
knowledge,  such  a model  has  not  been  proposed  for  application  to  patient 
Figure  4:  Radiotherapy  system  map  safety  to  date,  and  will  be  discussed  in  more  detail  later  in  this  report. 
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6.1  Case  study:  process  mapping 
and  conceptual  models  for  radiotherapy 

Before  we  could  proceed  to  model  the  radiotherapy  decision,  however, 
additional  work  was  required  to  specify  the  radiotherapy  system  itself,  and 
the  variables  in  that  system  that  may  be  subject  to  failure  and  therefore 
result  in  incidents.  The  qualitative  system  map  that  we  devised,  based  upon 
extensive,  iterative  discussion  among  the  co-authors,  is  represented  in 
Figure  4. 

The  main  path  (Assessment,  Preparation,  Treatment,  and  Follow-Up) 
represents  the  general  stages  of  care  that  occur  once  the  patient  enters  the 
treatment  facility.  Within  this  path,  specific  clinical  events  occur  which 
result  in  information  or  actions  performed  upon  the  patient.  Decision 
nodes  are  indicated  by  boxes.  Information  is  transferred  throughout  the 
process. 

Treatment  by  radiotherapy  typically  occurs  in  many  fractions  over 
the  course  of  several  weeks.  Although  this  system  map  is  specific  to 
radiotherapy  in  detail,  note  that  the  main  path  is  generalizable  to  many 
forms  of  medical  treatment.  The  map  is  hierarchical  or  layered  in  the 
sense  that  additional  details  can  be  represented  to  the  degree  necessary  to 
represent  the  specific  system  under  study  and  to  inform  the  decision.  The 
map  is  also  the  basis  for  quantitative  evaluation  of  the  benefits,  risks,  and 
costs  associated  with  radiotherapy. 

As  an  example  of  one  component  of  this  approach,  the  variables, 
decisions,  and  outcomes  associated  with  the  assessment  stage  (specific  to 
radiotherapy  for  breast  cancer)  can  be  represented  as  in  Figure  5 (see  next 
page). 

The  assessment  stage  is  represented  here  as  an  influence  diagram  (6)  in 
which  variables  that  have  uncertainty  associated  with  them  as  well  as  the 
potential  for  incidents  are  represented  as  ovals  (i.e.  probabilistic  nodes). 
These  variables  represent  the  different  sorts  of  information  that  are 
used  to  determine  the  “TNM”  (Tumor  size,  Nodal  status,  Metastasis) 
classification  (33)  for  a breast  cancer  patient’s  disease,  which  is  a common 
basis  for  determining  whether  radiotherapy  is  appropriate  for  that  patient. 
The  decision  to  conduct  imaging  and  the  decision  to  pursue  radiotherapy 
itself  are  represented  as  rectangles  influenced  by  the  probabilistic  nodes 
(arrows).  Influences  on  value  of  nodes  are  represented  by  dark  arrows, 
while  timing  and/or  structure  influences  are  represented  by  lighter  arrows. 
All  information,  tests  and  decisions  in  this  representation  have  associated 
costs  and  health  outcomes  (positive  and  negative),  represented  as  a 
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Timing  and/or  structure 
influences 


Figure  5:  Assessment  stage  influence  hexagon.  Potential  errors  can  occur  throughout,  but  ultimately  may  be 
diagram  reflected  in  the  decision  to  pursue  radiotherapy. 

This  influence  diagram  structure  is  important  because  it  is  not  only  a visual, 
qualitative  representation  of  important  decisions  at  the  beginning  of  the 
radiotherapy  process,  but  it  is  the  basis  for  a quantitative,  computational 
model  of  the  decision.  Influence  diagrams,  as  built  in  specialized  software, 
incorporate  calculations  that  are  similar  to  decision  trees,  which  are 
commonly  used  in  medical  decision  analysis  and  economic  evaluation  (29). 
These  methods  can  compare  across  different  alternatives  to  determine 
which  satisfy  the  objectives  in  better  ways  than  others.  Additionally, 
the  impact  that  different  sources  of  uncertainty  can  impose  on  making 
informed  decisions  can  be  assessed  through  sensitivity  or  value-of- 
information  analyses  (29). 

The  other  three  stages  in  Figure  4 can  be  represented  in  a similar  fashion. 
The  process  of  defining  the  variables  and  decisions  associated  with 
radiotherapy  has  been  an  interactive  and  challenging  exercise  for  the 
research  group.  As  in  most  complex  processes,  individuals  tend  to  work 
in  “boxes”  or  “silos;”  doing  their  jobs  without  necessarily  thinking  about 
what  other  individuals  or  departments,  units,  etc.  are  doing.  In  a complex 
care  pathway  such  as  this,  very  few,  if  any,  people  can  estimate  tradeoffs  and 
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rank  alternatives  “in  their  heads”  in  terms  of  preference.  This  may  or  may 
not  be  adequate  on  a day-to-day  basis,  but  it  certainly  invites  systematic 
failures  to  creep  in.  It  could  be  argued  that  the  process  of  individuals 
learning  from  other  individuals  is  one  of  the  most  valuable  components 
of  the  EBDM  process  as  applied  here.  This  is  precisely  why  these  tools  are 
being  developed;  to  help  inform  decisions. 

As  decisions  regarding  the  benefits,  risks,  and  costs  associated  with 
radiotherapy  are  hierarchical  and  complex,  and  we  are  in  the  process  of 
defining  some  of  the  decisions,  we  do  not  describe  all  the  details  of  modeling 
in  detail  here.  Rather,  we  describe  the  process  that  we  have  followed  in 
framing  the  decisions,  and  important  considerations  in  methodology. 


6.2  Decomposing  and  modeling  the  decisions 

6.2.1  UNCERTAINTY  ASSESSMENT 

We  have  expended  much  effort  on  different  aspects  of  decomposing  and 
modeling  the  decision  as  documented  in  Figure  1 (page  4).  For  example, 
the  assessment  influence  diagram  as  illustrated  in  Figure  5 shows  a 
range  of  different  types  of  tests  and  procedures  that  can  inform  the  TNM 
classification  decision  and  the  decision  to  pursue  radiotherapy  (for  breast 
cancer,  in  this  case).  Each  patient  may  not  require  all  of  the  tests  listed 
in  the  diagram.  Different  combinations  of  tests  and  procedures  will  be 
chosen  based  on  their  initial  presentation.  Certain  standards  are  in  place 
which  help  direct  the  proper  assessment  of  patients  while  at  the  same  time 
minimizing  risk  and  satisfying  resource  constraints.  By  modeling  different 
alternatives  (different  combinations  of  different  tests  and  procedures),  we 
can  start  to  estimate  which  combinations  are  most  beneficial  to  the  patient 
in  terms  of  proper  assessment,  which  are  associated  with  minimal  risk, 
and  which  satisfy  resource  constraints.  We  emphasize  the  critical  need  to 
examine  the  impact  of  uncertainty. 

As  an  example,  consider  the  information  value  and  potential  for  errors 
in  a pathology  review.  Pathology  information  is  critical  to  establishing 
the  “T”  and  “N”  components  of  TNM  classification.  The  literature  has 
shown  that  error  rates  of  1%  to  6%  can  occur  with  regard  to  pathology 
misdiagnosis  (34-36).  If  such  errors  are  not  caught  by  another  individual 
(i.e.,  another  pathologist,  oncologist),  then  they  can  be  potentially 
propagated  through  the  treatment  system  so  that  a decision  to  pursue 
radiotherapy  is  made  on  wrong  information,  thus  potentially  leading  to 
patient  harm.  At  another  level,  the  information  produced  by  a pathologist 
essentially  amounts  to  expert  opinion.  Assuming  the  same  biological 
specimen,  that  opinion  may  vary  across  pathologists,  and  indeed  within  the 
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Figure  6:  Concepts  of  accuracy  and 
precision 


same  pathologist  from  day  to  day  (the  degree  of  which  is  likely  dependent 
on  experience).  Thus,  even  if  the  information  is  not  wrong,  it  may  be 

uncertain,  and  that  uncertainty  can 
impact  a treatment  decision. 


PRECISION 


“Precision”  correlates  to 
“uncertainty”  here,  and  “accuracy” 
corresponds  to  “bias.”  For 
example,  a bathroom  scale  that 
on  average  displays  one’s  true 
weight,  but  which  varies  by  some 
amount  each  time  one  steps  on  it, 
is  imprecise  and  thus  its  display  is 
associated  with  uncertainty.  If  the 
scale  displays  one’s  weight  with 
little  variation  each  time,  but  the  dial  on  the  scale  is  set  artificially  low,  then 
the  display  is  precise  but  inaccurate,  and  thus  biased. 


Uncertainty  must  be  quantified 
in  order  to  determine  its  impact. 
Figure  6 illustrates  common 
statistical  representations  of 
uncertainty,  or  precision,  and 
accuracy,  or  bias. 


Accuracy  measures  the  ability  to 
achieve  an  objective,  specific  result. 
Precision  refers  to  the  ability  to 
repeatedly  achieve  a given  result. 


With  a bathroom  scale,  the  degree  of  uncertainty  and  bias  can  be  easily 
determined,  but  what  about  a subjective  variable  such  as  pathologist 
opinion?  Bayesian  statistical  methods,  which  have  been  used  by 
statisticians,  engineers,  and  other  quantitative  disciplines  for  hundreds  of 
years,  are  designed  to  quantify  subjective  uncertainty  (6).  As  an  example, 
say  that  five  pathologists  offer  their  opinion  as  to  the  probability  that  a 
particular  set  of  biopsy  slides  indicates  a particular  type  of  the  tumor.  Say 
that  their  opinions  are:  0.2,  0.3,  0.4,  0.5,  0.6.  If  this  is  all  the  information 
that  exists,  then  we  may  start  with  a uniform  distribution  (or  range  of  equal 
probabilities)  from  0.2  to  0.6.  This  is  what  is  known  as  a uniform  prior 
distribution.  This  distribution  may  change  if  we  recruit  more  pathologists 
(e.g.  five  more  who  all  say  0.3),  or  if  we  conduct  an  interactive  exercise  in 
which  all  the  pathologists  are  informed  of  the  initial  results.  The  art  and 
science  of  formal  expert  elicitation  is  often  employed  in  such  cases  (28). 

Scientists  and  physicians  who  are  trained  in  “classical”  or  “frequentist” 
statistics  find  this  uncomfortable,  because  frequentist  statistics  are 
designed  for  analyzing  data  sets,  not  quantifying  expert  opinion  or  multiple 
sources  of  uncertainty.  In  fact,  frequentist  statistics  can  be  viewed  as  a 
subset  of  Bayesian  statistical  decision  theory;  a subset  to  be  used  when  one 


’9 


QUANTITATIVE  APPROACHES  TO  PATIENT  SAFETY:  RESEARCH  IN  RISK  ANALYSIS  AND  RISK  MANAGEMENT  AS  APPLIED  TO  RADIOTHERAPY 


has  adequate  data  sets  (28).  In  many  decision  problems,  though,  like  the 
radiotherapy  issue  we  are  evaluating,  we  have  few  data  sets,  and  frequentist 
methods  are  not  useful  in  terms  of  informing  decisions.  Bayesian  methods 
that  are  designed  to  evaluate  highly  uncertain  scenarios  are  more  useful. 

Proper  evaluation  of  uncertainty  is  critical  for  many  different  reasons,  but 
the  most  critical  are: 

1.  determination  of  the  driving,  or  most  influential,  sources  of 
uncertainty  in  terms  of  the  results  of  a model  and  thus  a decision,  as 
well  as; 

2.  determination  of  the  reliability  of  a model  (28537). 

With  regard  to  the  first  reason,  a PRA  model,  and  our  proposed  PRADA 
model,  are  explicitly  designed  to  calculate  the  impact  that  each  variable  that 
goes  into  the  model  has  on  the  outcomes.  Referring  to  Figure  3,  there  will 
be  a set  of  uncertain  variables  that  represent  the  risks,  benefits,  and  costs 
associated  with  any  particular  risk  management  strategy.  Say  that  Strategy 
A is  current  practice,  and  Strategy  B and  C are  alternatives  designed  to 
reduce  risk  of  catastrophic  patient  harm.  One  way  to  calculate  the  impact 
of  uncertainty  is  to  simply  model  which  variables  impact  the  output  of  the 
model  the  most  (using  contribution  to  variance,  rank  correlation,  or  other 
statistical  methods);  but  ualue-of-injormation  is  a more  powerful  method. 

In  this  method,  the  mean  outcome  value  of  each  of  the  strategies  (in  terms 
of  benefits,  risks,  and  costs)  is  compared  by  running  all  of  the  model 
variables  through  the  model  using  all  their  possible  values.  The  differences 
between  the  outcomes  are  calculated  as  these  variables  are  varied  in  a 
simulation,  and  a value  for  each  of  the  variables  can  be  calculated  in  terms 
of  uncertainty  reduction.  In  other  words,  reduction  of  uncertainty  in  the 
most  uncertain  variables  will  be  most  valuable  in  terms  of  an  informed 
decision,  and  this  value  can  be  explicitly  estimated  in  dollars,  lives,  or 
whatever  measure  is  most  informative.  This  provides  direct  and  transparent 
information  to  a decision-maker  in  terms  of  where  uncertainty  reduction 
will  help  them  make  the  best  decision  (6). 

The  second  reason  for  proper  evaluation  of  uncertainty  in  a predictive 
model  relates  to  evaluation  of  the  reliability  of  the  model  and  its 
predictions. 

6.2.2  MODEL  RELIABILITY 

Reliability  is  defined  as  a measure  of  confidence  in  a model’s  predictions. 
Evaluation  of  the  reliability  of  model  predictions  involves  assessing  the 
magnitude  of  uncertainty  associated  with  model  predictions,  and  the 
acceptability  of  that  level  of  uncertainty  in  terms  of  decision-making  (37). 
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definition  of  context 


formulation  of  conceptual  model 


estimation  of  statistical  distributions 
and  parameter  values 


calculation  of  results 


sensitivity/ 

value-of-information  analysis 


documentation  and  communication  of  results 


Figure  j:  Factors  affecting  the  reliability  Factors  that  can  affect  the  reliability  of  model  predictions  are  represented  in 

of  model  predictions  Figure  7. 

Note  that  this  diagram  is  similar  to  Figure  1,  but  more  focused  on  the 
model  itself  rather  than  the  decision  process  as  a whole. 

Each  of  the  factors  in  the  figure  can  be  associated  with  different  sorts  of 
uncertainty,  or  bias.  For  example,  if  an  inappropriate  question  is  asked,  an 
inappropriate  answer  will  result.  Uncertainty  analysis  is  usually  targeted 
toward  estimation  of  statistical  distributions  to  be  used  in  the  model,  but 
can  also  be  applied  to  the  model  itself  by  examining  the  impact  of  different 
model  structures  on  results. 

In  the  present  project,  model  reliability  cannot  be  completely  determined 
because,  to  our  knowledge,  catastrophic  events  have  not  occurred  at  the 
TBCC.  In  other  words,  our  model  results  will  always  be  hypothetical  to 
some  degree  until  such  events  occur  (and  of  course  we  hope  that  they  never 
do).  This  is  not  unlike  famous  failure  scenarios  such  as  space  shutde  and 
nuclear  power  plant  disasters,  and  extreme  acts  of  terrorism  such  as  “9-11.” 
A national  incident  report  system  would  of  course  be  useful,  but  it  remains 
to  be  seen  whether  and  how  such  information  will  be  useful  for  the  purpose 
of  quantitative  modeling.  Regardless,  the  probability  and  consequences 
of  such  incidents  can  still  be  predicted  by  characterizing  and  modeling 
components  of  the  system  and  thus  managing  risks,  but  the  results  will  be 
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associated  with  a high  degree  of  uncertainty.  The  key  to  a reliable  model  is 
capturing  and  quantifying  this  uncertainty  to  the  degree  possible,  and  then 
examining  the  impact  on  decisions.  Decision-makers  who  are  presented 
with  single  estimates  of  the  probability  of  rare,  catastrophic  events  are 
essentially  being  deceived! 

Decisions  involving  management  of  catastrophic  events  are  of  course 
not  only  informed  by  probabilities,  but  by  a number  of  other  factors.  Two 
considerations  that  are  important  in  risk  management  are  risk  aversion, 
and  the  tradeoffs  associated  with  resource  constraints  and  other  important 
factors. 

6.2.3  RISK  AVERSION 

Different  levels  of  aversion  to  risk  exist  in  any  decision  situation,  according 
to  the  situation  itself  and  dependent  on  the  individuals  involved.  Different 
levels  of  risk  aversion  are  shown  in  Figure  8. 

These  curves  represent  utility  functions,  utility  representing  the 
amount  of  value  that  different  people  would  place  on  different 
outcomes,  given  a risky  gamble  or  bet  (38).  In  the  case  of  the  figure, 
a risk-averse  physician  may  place  decreasing  value  on  incremental 
increases  in  life  expectancy  given  an  incrementally  risky  procedure; 
that  is  the  physician  may  bet  that  she  can  give  the  patient  one  year 
of  life  expectancy,  but  probably  not  ten  years  if  the  risk  is  higher. 
Conversely,  a risk-loving  physician  may  bet  that  she  can  give  the 
patient  ten  years  with  a highly  risky  procedure.  Surprisingly  little 
empirical  work  has  been  done  with  healthcare  providers  and 
organizations  (39);  but  depending  on  the  discipline,  they  likely  span 
a wide  spectrum  of  risk  aversion.  In  another  example,  the  business 
of  high-risk  financial  trading  where  risk  is  defined  as  incorporating  a 
two-tailed  distribution  of  consequences  (one  can  win  big  or  lose  big) 

Figure  8:  Risk  aversion  curves  likely  attracts  risk-neutral  to  risk-loving  people;  risk-averse  people  would 

get  ulcers! 

Government  organizations  such  as  environmental  and  traffic  safety 
agencies  tend  to  be  risk-averse:  they  err  on  the  side  of  safety.  Regardless, 
it  is  important  to  capture  risk  aversion  in  any  decision  scenario,  as  the 
outcomes  of  risk  management  decisions  may  vary  radically  depending 
upon  the  individual  or  organization  involved,  as  well  as  the  context.  For 
example,  the  ancillary  risk  of  litigation  in  the  US  healthcare  system  is  much 
higher  than  the  risk  in  the  Canadian  system;  thus  risk-averse  behavior  may 
be  more  prevalent  in  US  organizations  and  risk  management  practices  to 
date  may  have  differed  (but  this  has  not  been  empirically  studied,  to  our 
knowledge). 
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In  the  present  project,  we  are  explicitly  including  risk  aversion  in  assessing 
different  alternative  risk  management  strategies.  As  risk  aversion  is 
incorporated  into  the  concept  of  utility,  we  feel  it  is  important  to  assess  the 
sensitivity  of  decisions  to  different  degrees  of  risk  aversion. 

6.2.4  MULTI-ATTRIBUTE  UTILITY 

Risk  aversion  for  a single  utility  outcome,  life  expectancy,  is  illustrated  in 
Figure  8.  Utility  outcomes  as  decision  objectives  are  quantified  by  attributes; 
in  this  case,  the  attribute  is  years  of  life  gained  as  a result  of  some  medical 
intervention  (30).  A decision  in  healthcare  usually  involves  some  level 
of  maximizing  years  of  life,  degree  of  health,  etc.  However,  a procedure 
that  increases  longevity  may  not  necessarily  increase  quality  of  life;  thus  a 
tradeoff  exists. 

There  are  other  important  considerations  in  healthcare.  Healthcare  costs 
money  and  takes  manpower,  therefore  there  are  resource  constraints. 

There  may  be  yet  other  constraints  that  would  limit  choices,  such  as  laws 
or  union  rules.  There  may  be  other  utility  outcomes  that  are  important 
to  providers,  such  as  maximizing  physician  or  nurse  satisfaction  within 
an  organization  and  with  working  conditions.  These  sorts  of  utility  and 
constraint  considerations  need  to  be  accounted  for  in  important  decisions, 
but  they  add  a measure  of  complexity,  because,  in  nearly  all  cases,  a gain 
in  health  utility  results  in  a loss  of  something  else.  This  introduces  the 
concept  of  multi -attribute  utility,  in  which  different  objectives  are  traded  off 
against  each  other,  with  the  intent  of  finding  an  alternative  that  satisfies  the 
most  objectives  in  the  best  way  possible. 

For  example,  say  we  want  to  maximize  the  health  benefit  of  radiotherapy, 
minimize  the  risk  of  patient  harm,  and  stay  within  financial  constraints. 
Health  benefit  is  commonly  measured  by  quality  adjusted  life-years  (QALYs) 
gained,  which  is  generally  viewed  as  a preferable  measure  to  “raw”  life 
expectancy  in  that  it  can  partially  correct  for  the  tradeoff  mentioned 
above  (28).  Minimizing  patient  harm  can  also  be  measured  by  QALYs, 
or  on  another  scale.  Financial  constraints  can  be  measured  by  dollars. 

All  attributes  that  quantify  these  objectives  can  be  incorporated  into  a 
multi-attribute  utility  function.  This  function  can  be  used  in  a decision 
model  (e.g.  going  back  to  Figure  3)  to  compare  across  alternatives.  Single 
attributes,  such  as  QALYs  or  money,  can  be  assessed,  but  in  a complex 
healthcare  decision  these  do  not  tell  the  whole  story.  Tradeoffs  exist,  so 
they  should  be  assessed. 

A common  means  of  assessing  the  tradeoffs  across  health-related  utility 
and  resource  constraints  is  the  cost-effectiveness  ratio;  that  is  what  is  the 
incremental  cost  per  unit  of  health  utility  gained  by  one  alternative 
compared  to  another?  There  is  a rich  literature  on  cost-effectiveness 
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analysis  (e.g.  references  in  (29)),  and  in  many  cases  cost-effectiveness 
can  be  quite  informative  (say,  in  making  allocation  decisions  across 
similar  types  of  interventions  in  the  same  department  of  a healthcare 
organization).  However,  there  are  many  conceptual  and  theoretical 
problems  associated  with  cost-effectiveness  as  a decision  criterion  (40-42); 
thus  we  have  chosen  to  use  more  rigorous  and  generalizable  multi-attribute 
methods.  Multi-attribute  outcomes  are  also  easily  converted  into  measures 
of  net  benefit  (as  in  benefit-cost  analysis)  if  this  seems  to  be  informative. 

6.3  Where  are  we  now? 

We  are  now  in  the  process  of  risk  and  decision  modeling  as  described 
above.  This  particular  project  is  expected  to  be  completed  in  2006, 
and  is  expected  to  inform  decision-makers  with  regard  to  risk  and  risk 
management  as  applied  to  radiotherapy  in  the  TBCC  as  our  first  case  study. 
However,  we  are  exploring  simultaneous  and  subsequent  projects  as  part 
of  a larger  research  program.  Some  of  these  projects  are  in  Section  7.3 
(page  27). 
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Additional  Research 


7.1  Resource  allocation  considerations 

Radiotherapy  programs  employ  a variety  of  quality  control  procedures  not 
only  aimed  at  assuring  that  currently  accepted  treatment  tolerances  are 
routinely  met  but  also  to  act  as  filters  in  identifying  major  incidents  that 
may  occur  in  the  complex  series  of  processes  involved  in  the  treatment 
of  a patient.  Although  such  procedures  consume  significant  resources  in 
any  advanced  radiotherapy  program,  the  distribution  of  these  resources 
is  generally  based  on  historical  evolution  over  many  years  rather  than 
a systematic  and  quantitative  study  of  possible  incidents  in  the  total 
radiotherapy  process.  We  believe  that  a PRADA  model  of  radiotherapy, 
such  as  discussed  in  Section  6,  can  lead  to  a more  effective  distribution  of 
those  quality  control  resources  targeting  the  avoidance  of  rare,  catastrophic 
incidents. 

The  resource  implications  of  alternative  risk  management  strategies  is  the 
issue.  The  resources  are  almost  entirely  human  and  thus  knowledge  of  the 
time  taken  and  remuneration  of  the  various  professional  groups  involved 
constitute  a reasonably  accurate  basis  for  the  evaluation  of  resource 
consumption.  An  activity-based  costing  approach  is  an  appropriate  means  of 
doing  this  (43). 

All  those  interventions  that  could  lead  to  a reduction  in  the  probability  and 
severity  of  untoward  events  concerning  patients  need  to  be  evaluated.  These 
include,  for  example,  pathology  review  rounds,  quality  control  of  treatment 
equipment,  redundant  checking  of  laboratory  reports  and  treatment  plans. 
In  many  cases  such  activities  take  place  routinely  and  estimating  their 
costs  is  relatively  straightforward.  Technological  advances,  such  as  the 
introduction  of  multi-leaf  collimation  in  radiotherapy,  also  have  resource 
implications  as  well  as  potential  safety-related  benefits(44).  Any  risk  and 
decision  model  that  does  not  become  quickly  obsolete  will  need  to  be 
flexible  enough  to  accommodate  such  future  developments. 

It  is  likely  that  computational  risk  models  such  as  those  discussed  here  will 
suggest  alternative  strategies  for  risk  management  that  are  not  currently 
implemented  at  the  Tom  Baker  Cancer  Centre  or  elsewhere.  Estimating 
the  cost  of  such  strategies  is  associated  with  more  uncertainty  than  those 
of  existing  procedures.  Thus,  clearly  the  acknowledgement  of  uncertainty 
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in  cost  estimates  is  as  essential  as  those  in  risk  estimates  and  comprises 
a significant  component  of  any  systematic  study  of  patient  safety  and  risk 
management. 


I 


7.2  Incident  learning 

Previous  research  by  Cooke  (45)  and  others  suggests  that  while  accidents 
in  complex  socio-technical  organizations  are  normal,  risk  can  be  reduced 
by  the  implementation  of  an  effective  management  system  for  learning 
from  incidents.  Such  systems  have  been  successfully  implemented  in 
high-risk  industries  such  as  chemicals  manufacturing  and  aviation,  but 
their  potential  to  reduce  accidents  in  the  healthcare  industry  has  not  been 
fully  explored.  McFadden  et  al.  (46)  have  suggested  that  the  critical  success 
factors  for  controlling  and  managing  hospital  incidents  will  be  similar  to 
those  found  in  the  aviation  industry. 

One  hypothesis  is  that  patient  safety  in  a complex  healthcare  system  can  be 
improved  by  implementing  an  effective  system  for  learning  from  incidents. 
Such  a system  will  act  as  a continuous  improvement  process,  potentially 
reducing  the  probability  and  severity  of  incidents  and  leading  to  improved 
patient  safety  and  more  reliable  healthcare  performance  over  time. 

Research  questions  that  may  be  addressed  include  the  following: 

1.  What  information  on  actual  and  potential  incidents  is  required  to 
characterize  those  incidents  and  how  might  such  information  most 
effectively  be  collected? 

2.  How  can  the  information  so  obtained  be  collated  and  analyzed  so  that 
it  can  be  fed  back  into  the  patient  care  system? 

3 . Does  a system  for  learning  from  incidents  lead  to  significant 
improvements  in  measures  of  patient  safety  and  operational 
effectiveness? 

Our  proposed  incident  learning  system  will  be  fully  aligned  with  the 
process  model  under  development  (Figure  3,  page  13).  The  proposed 
incident  learning  process  will  enhance  and  extend  the  decision  analysis 
approach  described  earlier  by  providing  a framework  for  gathering  incident 
information,  disseminating  knowledge  from  the  incident  investigation, 
enabling  corrective  actions  to  be  taken  and  informing  risk  management 
decisions.  The  underlying  quantitative  nature  of  this  approach  will  greatly 
facilitate  objective  assessment  of  the  effectiveness  of  the  incident  learning 
system. 

The  methodological  approach  will  be  similar  to  that  of  McAfee  (47)  who 
studied  the  impact  of  enterprise  information  technology  adoption  on 
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operational  performance  at  a manufacturing  company.  This  project  will 
employ  a case  study  design  and  a natural  experiment.  The  case  study  design 
will  document  typical  radiation  oncology  and  medical  physics  operations 
before,  during,  and  after  the  implementation  of  an  incident  learning 
system.  The  natural  experiment  will  compare  incident  data  and  other  safety 
metrics  for  a period  before  system  implementation  to  similar  data  post- 
implementation. There  will  be  an  action  research  component  to  the  project 
because  Dr.  Cooke  will  be  advising  on  system  design  and  implementation. 
Westbrook  (48)  advocates  action  research  as  an  approach  that  may 
be  relevant  to  practitioners  “because  real  world  operations  often  face 
unstructured  problems,  which  cannot  be  modeled,  but  must  be  managed.” 

Expected  outcomes  from  this  component  of  the  project  will  be  a case  study 
to  inform  decision-makers  in  healthcare  with  regard  to  the  value  of  an 
incident  learning  system  and  an  agreed-upon  design  for  implementation  of 
such  a system.  The  overall  incident  learning  project  is  expected  to  take  2-3 
years  to  implement  locally,  with  the  first  year  being  focused  on  the  design 
of  methods  for  the  acquisition,  analysis  and  feedback  of  information.  If  the 
incident  learning  case  study  and  pilot  project  is  judged  to  be  of  value  locally, 
it  is  expected  that  similar  systems  would  be  suitable  for  implementation  at 
other  provincial  healthcare  institutions  and  ultimately  at  a national  level. 


7.3  Future  projects 

The  previous  discussion  focuses  on  projects  that  are  ongoing.  Productive 

areas  of  research  that  are  part  of  a larger  research  program  include: 

• Establishing  the  value  of  the  model  over  time  in  terms  of  decision 
facilitation  at  the  TBCC,  and  modification  of  the  model  as  appropriate. 

• Explicitly  addressing  patient  values,  objectives,  and  risk  aversion  in  the 
analysis. 

• Use  of  continuous-risk  utility  assessment  for  stakeholder  values. 

• Evaluation  of  the  reliability  of  the  PRA  model  by  implementing  a 
tracking/reporting  system  for  failures. 

• Establishing  the  reliability  of  the  model  over  time  in  longitudinal  studies 
in  terms  of  reduction  of  probability  and/or  consequences  of  failures. 

• Developing  and  applying  operational  simulation  models  (integrated  with 
the  PRADA  model)  to  inform  optimization  of  treatment  strategies. 

• Generalizing  the  approach  to  all  forms  of  cancer  treatment  conducted  in 
the  TBCC.  In  particular,  a comparative  analysis  of  the  risks,  benefits,  and 
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costs  of  radiation  oncology  as  compared  with  surgery  and  chemotherapy, 
as  well  as  combined  approaches,  will  be  informative. 

• Evaluation  of  changes  to  technology  and  the  resultant  impact  on  the 
potential  for  failures.  Radiation  therapy  technologies,  like  most  areas  of 
medicine,  are  rapidly  changing;  the  flexible  framework  presented  here 
will  be  amenable  to  evaluations  of  the  impact  of  new  technologies. 

• Conducting  stakeholder  (especially  patients)  surveys  and  elicitation 
exercises  to  refine  outcome  dimensions,  using  methods  such  as 
contingent  valuation  and  willingness-to-pay/accept. 

• Exploration  of  learning  as  a result  of  use  of  the  PRADA  model  by 
employing  statistical  process  control  methods. 

• Exploring  stochastic  effects  (i.e.  cancer)  that  are  potentially  associated 
with  radiotherapy. 

• Development  and  dissemination  of  a user-friendly  spreadsheet  tool  to 
facilitate  use  of  the  approach  by  analysts  and  decision-makers. 

• Applying  the  approach  to  non-cancer  healthcare,  for  example  intensive 
care,  pharmacy,  etc.  Interest  in  such  an  approach  has  been  already 
expressed  by  Department  heads  and  senior  management  in  the  Calgary 
Health  Region,  an  integrated  health  region  serving  1.3  million  people 
with  an  annual  budget  of  $1.8  billion.  Application  of  the  approach  for 
informing  patient  safety  decisions  in  such  a large  healthcare  setting 
would  potentially  have  large  implications  for  patient  care  and  resource 
allocation. 

• Applying  the  approach  in  non-Canadian  healthcare  settings.  Examples 
may  be  private  systems  such  as  US  health  maintenance  organizations, 
and  publicly  funded  systems  such  as  the  US  Veteran’s  Administration, 
the  United  Kingdom,  the  European  Union,  and  Australasia. 
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Opportunities 


In  this  report  we  have  discussed  the  issues  and  reviewed  the  current  state- 
of-the-art  in  proactive  structured  patient  safety/risk  management  programs. 
We  see  a major  role  for  quantitative  methods  in  this  vital  area  of  health 
systems  operations  and  have  suggested  what  form  such  methods  might 
take.  In  concluding,  we  further  suggest  that  an  appropriate  model  of  the 
type  described  here  for  the  radiotherapy  case  study,  coupled  with  a rigorous 
analysis  and  decision  framework,  could  inform  a comprehensive  patient 
safety/risk  management  program  in  many  areas  of  healthcare. 
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